Adds IndexedNearestJoinExternal alongside the existing IndexedNearestJoin (Lance R)
and the per-query temp-Lance path (#3). When R is a direct parquet/Delta scan,
build a Lance external IVF-PQ index over the source files and probe + refine +
post-topK materialize from those files — no temp Lance dataset, no column copies.

Sibling RFC + benchmark numbers:
  #4

== Files ==

internal/ExternalIndexProbe.scala
  Scala wrapper around the new ExternalIvfPqIndex JNI handle (in lance-core
  7.1.0-beta.1). Decodes Arrow IPC stream from fetchRows into Spark-compatible
  row maps.

internal/ExternalIndexLifecycle.scala
  Driver-side cache (sha256 keyed by file paths + col + params); cleanup hook
  registered with the existing LanceTempLifecycle so cluster scratch is cleaned
  on application end / JVM shutdown — same machinery as the temp-Lance path.

internal/ExternalFusedStage.scala
  Single-stage probe + materialize. Lance returns global top-K per query already
  (no cross-task merge needed), so the leftId shuffle from the Lance-native
  pipeline is vestigial here. Fused stage opens the index once per task, probes
  every left row, batched fetchRows for the partition, emits final join Rows.

internal/LanceVectorIndexBuilder.scala
  Promoted from src/test/scala — needed by IndexedNearestJoinExternalBenchmark
  to build IVF-PQ on Lance-native R for the C-indexed apples-to-apples reference
  config. Same body as the existing test helper.

  Note: fixes a latent arg-order bug in the existing test helper —
  VectorIndexParams.ivfPq signature is (numPartitions, numBits, numSubVectors,
  ...), not (numPartitions, numSubVectors, numBits, ...). The bug was masked
  in the existing test because its defaults set both args to 8 (swap invisible).

IndexedNearestJoinExternal.scala
  Public DataFrame API entry point. SQL Catalyst integration deferred to Phase 3.

benchmark/IndexedNearestJoinExternalBenchmark.scala
  Compares B-narrow / B-wide (temp-Lance, project rid only / +N payload) vs
  C-indexed-narrow / C-indexed-wide (Lance-native + IVF-PQ index, same projections)
  vs E (external-index over parquet). Reports E warm vs C-indexed (the apples-
  to-apples comparison: same algorithm, R lives in parquet vs Lance).

  Headline cluster numbers at wide-medium (|R|=1M, dim=128, 16 string payload
  cols, K=10, |L|=100):
    B-narrow:   18,128 ms
    E warm:        570 ms  (E build amortizes after ~2 queries vs B-narrow)

benchmark/IndexedNearestJoinTempRBenchmark.scala
  Drive-by fix: pre-set spark.lance.knn.tempR.dir from BENCH_DATA_PATH in
  cluster mode so config D doesn't trip the helper's cluster-mode fail-fast guard.

== Lance bump ==

pom.xml: lance.version 6.0.0-beta.4 → 7.1.0-beta.1 — required for the new
ExternalIvfPqIndex API. Companion lance-core PR:
  sezruby/lance#1

== Tests ==

IndexedNearestJoinExternalTest passes locally:
  16 left vectors × 2 parquet files of 320 rows; top-K returned per left;
  scores monotone; oracle hit rate above K/2 threshold.

Three fixes to the external-index path based on cluster benchmark findings.

== 1. Parallelism heuristic ==

ExternalFusedStage previously ran with `left.rdd.partitions.size` tasks. For
typical KNN workloads |L| is small (hundreds to thousands of queries) and Spark
right-sizes left to 1-2 partitions accordingly, leaving fetchRows running
serially on a 32-core cluster.

IndexedNearestJoinExternal.apply now sizes parallelism from source row count:

  targetRowsPerTask = 100  // ~5-50ms per Lance search × 100 = 0.5-5 sec/task
  estimatedRows     = left.queryExecution.optimizedPlan.stats.rowCount
  sizedParallelism  = ceil(estimatedRows / targetRowsPerTask), capped at
                      defaultParallelism
  parallelism       = mergeParallelism.getOrElse(sizedParallelism)

Conditional repartition: only UP, never DOWN. Users with already-good
partitioning aren't slowed down.

Also adds diagnostic prints so benchmarks see what parallelism the path picked.

Cluster impact at wide-medium (|R|=1M, |L|=100, 32-core cluster):
  Before: 1-2 tasks → ~4,300 ms warm
  After:  32 tasks  → ~600 ms warm  (7× speedup)

== 2. Bench scratch cleanup ==

Cluster scratch volumes accumulate 5-10 GB per bench run and eventually trip
"Disk quota exceeded" mid-write. Bench now sweeps stale `knn-bench-data-*`
siblings from dataRoot.getParent at start. Strict pattern match (only matches
the cpd-submit-bench naming), skips this run's own dataRoot.

== 3. mega-medium opt-in ==

`mega-medium` scale (|R|=10M) is now documented as requiring substantial scratch
volume; default scale lists exclude it.

Cluster bench results at wide-medium (medians across 4 runs):
  B-narrow (temp-Lance):       16,506 ms
  C-indexed-narrow (Lance-R):     545 ms
  E warm (external-index):        609 ms
  E build (one-time):          29,529 ms

E warm comparable to C-indexed-narrow (both sub-second); E ~27× faster than
B-narrow. E build amortizes after ~2 queries vs B-narrow.

Adds LanceParquetIndex, a SparkSession-aware wrapper around ExternalIvfPqIndex
that exposes single-query and small-batch retrieval shapes (RAG / point-lookup
KNN) alongside the existing IndexedNearestJoinExternal bulk join.

The Rust core and JNI surface already supported single-query search; this
commit exposes it from Scala without forcing callers through kNearestJoin
with a 1-row left DataFrame (which is bounded below by Spark task launch
latency).

Surface:
  - LanceParquetIndex.build / buildIfMissing / open
  - search(query, k, ...) — driver-side, returns Seq[SearchResult]
  - searchToDF(query, k, projection) — 1-partition DataFrame for pipeline use
  - fetchRowsToDF(refs, projection) — random-access fetch with payload
  - buildIfMissing reuses ExternalIndexLifecycle's content-hash cache so
    the index file is shared with the bulk-join path

LanceParquetIndexTest covers JNI round-trip, DataFrame schema shape,
projection materialization, fetchRows ordering, and cache reuse. Recall is
tested in the Rust-side external_index_phase1.rs; here we focus on what the
wrapper itself can break.

Adds ExecutorCpuCheck.run() which executes a fixed-cost CPU loop on every
task slot, groups results by executorId, and prints a per-executor median /
min / max table sorted slow-to-fast. The probe runs once before the first
scale and answers the question "are these executors actually free?" — which
defaultParallelism alone cannot.

Output highlights the slowest/fastest ratio. ≤1.25× is uniform; ≥1.5×
prints a warning that bench medians are under contention. With
BENCH_CPU_CHECK_FAIL_RATIO set the probe throws if the ratio exceeds the
threshold, refusing to run a noisy bench.

Skip with BENCH_CPU_CHECK_SKIP=true when the bench is being used for
something other than measuring (smoke tests, debugging).

Existing benchmark output gains a new section before any timing happens, so
post-hoc readers can see whether the numbers were collected on a quiet
cluster or a contended one.